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Indian Standard INFORMATION TECHNOLOGY -- GUIDELINE FOR THE EV'UATION AND SELECTION OF CASE TOOLS

1

Scope

This International Standard deals with the evaluation and selection of CASE tools, covering a partial or fill portion of the software engineering life cycle. It establishes processes and activities to be applied for the evaluation of CASE tools and selecting the most appropriate CASE tools from several candidates. These processes are generic, and organizations must tailor them to meet organizational needs. The CASE tool evaluation and selection processes should be viewed in the larger context of the organization's technology adoption process. This International Standard provides: a. b. Guidance on identi~ng organizational requirements for CASE tools.

Guidance on mapping those requirements to CASE tool characteristics to be evaluated. A process for selecting the most appropriate CASE tool fiorp several tools, based on measurements of the defined characteristics.

c.

Primary users of this International Standard are organizations that intend to adopt CASE tools to support their soilware life cycle processes. CASE tool suppliers may also use this International Standard to describe characteristics of their CASE tools. This International Standard is not intended to apply to: a. Software engineering frameworks whose purpose is to provide mechanisms for data, control and presentation integration.
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IS 14653:1999 ISO/IEC 14102:1995 b. General purpose tools (e.g., word processors, spreadsheets) which may be used in sotlware engineering activities, nor CASE tools of very narrow scope or specific purpose (e.g., a compiler). Planning for the implementation of CASE tools within an organization (even though it is recognized that this is an important subject).

c.

Standard may make the best possible selection of a CASE NOTE- A user of this International of a successful implementation. ISO/IEC JTC 1 SC7 WG4 is working on a toolandhavenoguarantee draft technical report, Adoption of CASE Tools, which addresses this subject.

This International Standard contains a set of processes, activities, and tasks designed to be tailored. The tailoring process is the selection of applicable processes, activities and tasks. Compliance with this International Standard is defined as the petiormance of the processes, activities, and tasks selected from this International Standard for the evaluation and selection project. Any organization imposing this International Standard as a condition of trade is responsible for speci~ng the minimum set of required processes, activities, and tasks which constitute compliance for a given application of this International Standard. Defining and documenting that specification forms part of the initiation process (clause 5).
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2

Normative references

The following standards contain provisions which, through reference in this text, constitute provisions of this International Standard. At the time of publicatio~ the edkions indicated were vfl]d. AU standards are subject to revisio~ and parties to agreement based upon this International Standard are encouraged to investigate the possibility of applying the most recent editions of the standards indicated

below. Members of IEC and 1S0 maintain registers of currently vdld International Standards.
1S0 5807:1985, Information processing-Documentation symbols and conventions for alzta, program and system flowcharts, program network charts and system resources charts.

ISOIIEC 12119:1994, Information technology - Software packuges - Quality
requirements and testing. K30AEC 12207:1995, Information technolo~ - Software ll~e cycle processes.

ISO/IEC 9126:1991, Information technolo~ - Software product evaluation Quality characteristics andguidelinesfor their use.
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3
3.1

Definitions and acronyms
Definitions

For the purposes of this International Standard, the following definitions apply.
An action of applying specific documented criteria to a 3.1.1 assessment: specific software module, package or product for the purpose of determining acceptance or release of the soflware module, package or product. (ISO/I.EC 9126:1991) 3.1.2 atomic subcharacteristic:

The highest level evaluation categories are called characteristics. Characteristics are usually subdivided into subcharacteristics. Many subcharacteristics may be fi.u-thersubdivided into lower level subcharacteristics, At the lowest-level, when no firther subdivision is appropriate, the subcharacteristics are referred to as atomic subcharacteristics.
3.1.3 CASE tool:

A software product that can assist software engineers by providing automated support for soilware life-cycle activities as defined in ISO/IEC 12207:1995.

NOTES 1- A CASE tool may provide support in only selected functional areas or in a wide variety of tinctional areas. 2- CASE tools maybe used in several modes: . As stand alone tools; in this case, only compatibility with environment elements should be addressed.
q In small groups which communicate directly with one anothe~ it may be supposed that integration is predefine, perhaps propnetorily.

q In the presence of a larger framework of the SEE; in this case the ability of the tool to use the relevant services of the framework should be addressed.

3.1.4 characteristic:

An aspect of a product by which it can be described and

evaluated. A characteristic maybe refined into multiple levels of subcharacteristics that bear on its ability to satis~ stated or implied needs.
3.1.5 measurement: The action of applying a soflware quality metric to a specific software product. (ISO/IEC 9126:1991)
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IS 14653:1999 ISO/IEC 14102 :199S NOTES 1- Measurement can apply to metrics other than software quality metrics. 2- h object maybe measured directly, or may be measured indirectly by the application of metricsto information about or representations of the object.

3.1.6 metric:

A quantitative scale and method which can be used to determine the value a subcharacteristic takes for a specific software product.
3.1.7 rating:

The action of mapping the measured value to the appropriate rating level. Used to determine the rating level associated with the software for a specific quality characteristic. (ISO/IEC 9126:1991)

NOTE - Rating and rating levels can be applied to characteristics other than quality characteristics.

3.1.8 rating level:

A range of values on a scale to allow sofiware to be classified (rated) in accordance with the stated or implied needs. Appropriate rating levels may be associated with the different views of quality, i.e., users, managers or developers. These levels are called rating levels. (ISO/IEC 9126:1991)
3.1.9

Software Engineering Environment: The soflware engineering environment (SEE) is that portion of the system which provides automated support for the engineering of sollware systems and the management of the software process. It includes platform, system software, utilities, and CASE tools installed.
NOTE - The SEE architecture has two aspects:
q

the CASE tools which provide facilities for supporting life-cycle processes, and a general framework which provides a set of capabilities that offer common services used by

&e tools.

3.2

Acronyms

BMT CASE GUI SEE SQL

Bench Mark Test Computer Aided Software Engineering Graphical User Interface Software Engineering Environment Structured Query Language

9
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4

Overview of evaluation and selection of CASE tools

This section provides an overview of the evaluation and selection of CASE tools discussed in this International Standard as shown in Figure 1. Evaluation and selection of CASE tools includes four major processes: Initiation Process Structuring Process Evaluation Process Selection Process
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Figure 1- Overview of evaluation and selection of CASE tools
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A key process is the structuring of a set of requirements against which candidate CASE tools are to be evaluated, and upon which selection decisions will be based. The CASE tool characteristics defined in clause 9 form the basis for requirements structuring, and play a central role in the overall process. 4.1
Initiation process

The purpose of the initiation process is to define the general objectives and requ~ements of the intended evaluation and selection of CASE tools, to `establishthe high level direction, and to define the management aspects of the effort (e.g., schedule, resources, cost). The initiationprocess, discussed in detail in clause 5, is composed of three activities:
q

Ii@ setting: Provides the ration~e and %nerd PolicYfor ev~uation and
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selection.
q

establishing selection criteria: provides criteria to be used in the subsequent

selection process.
q

results in a plan which includes generic planning information and also information which defines the structure of the evaluation and selection effort.
project planning: 4.2 Structuring process

The purpose of the structuring process is to elaborate a set of structured requirements, based upon the CASE tool characteristics of clause 9 against which CASE tools should be evaluated, and to obtain the necessary information on CASE tools to permit evaluation. It is assumed that a set of general organizational information and guidelines is available to be used as inputs. The structuring process, discussed in detail in clause 6, is composed of three activities:
q

requirements

analysis: transforms organizational needs into measurable

structures.
CASE tool information gathering: captures a snapshot of the current stateof-the-art in CASE tools.
q q

identifying final candidate CASE tools: candidate CASE tools are identified

for evaluation using the results of the last two activities.
NOTE - During the evaluation, requirements may require revision. of this, and subsequent processes maybe necessary. activities

If this occurs,somerepetitionof

4.3

Evaluation process

The purpose of the evaluation process is to produce technical evaluation reports that will be the major input for the selection process. Each evaluation process results in a profile of the quality and other characteristics of the tool which was evaluated. Comparisons between tools are not made as part of this process. The evaluationprocess, discussedin detail in clause 7, is composed of three activities:
q

preparation

for evaluation:

finalization of the various details of the
11
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evaluation (e.g., scenario, subcharacteristics, metrics, tool characteristics) in an evaluation plan.
q

evaluating CASE tools: measurement, rating and assessment. evaluation reporting: an evaluation report is prepared which provides the

q

results of the evaluation for each CASE tool considered. 4.4
Selection process

The purpose of the selection process is to identifi the most suitable CASE tool(s) among the candidate tools, and to ensure that the recommended tool(s) meets the original goals. The selection process compares the results of the evaluations of the candidate tools to determine which is the most appropriate for selection. The selection process, discussed in clause 8, is composed of four activities:
q

preparing for selection: the selection criteria are finalized and the selection algorithm is defined.
q

assessing the evaluation results: the selection algorithm is applied to the

evaluation results.
q

recommending a selection decision: the best of the candidates is determined. validating the selection decision: the recommended selection is validated

q

against the original goals. 4.5
General process considerations

There are several considerations that apply to the processes described in this International Standard on a global basis. The intent is for the user of this International Standard to tailor its application in such a way as to maximize the probability of a successfid evaluation and selection process, and minimize its cost and risk.
4.5.1 Sequencing of processes

This International Standard does not impose the sequence of process activities describ~d above and in the following sections. It is up to the organization to select the relevant processes and activitiesneeded to meet its evaluation and selection goals. The organization will decide which to employ, in what sequence, and with what

12
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degree of parallelism. The sequencingof the processes' activities is then documented in the project plan developed during the initiation process.
4.5.2 Reducing cost and risk

In general, organizations which apply this International Standard will want to minimizethe cost of the entire evaluationand selection process to the extent possible, while maintaining the level of effort necessary to select the most appropriate CASE tool(s) for their use. These objectives may be addressed by minimizing the number of tools evaluated, minimizingthe cost of evaluating specific tools, and ensuring that the formality of the process ii appropriate to the organization.
The activities of CASE tool information gathering and identifying final candidates for selection (see clause 6) effectively allow the user of this International

Standard to screen the available tools against the organization's needs, and eliminate from consideration tools which do not, or are not likely to, substantially address the organization's needs.
NOTE 1 - It may be that the organization is unable to find any tool which appear likely to sutliciently meet its needs. In such a case, the stated needs themselves should be re-examined, and if they are found to accurately reflect the organization's actual requirements for technology improvement the overall evaluation and selection process may be abandoned. Similarly, if the final candidate tools appear to be marginal in addressing the organization's needs, the level of detail and formality of the subsequent activities should be made to reflect the risk factor, and the organization should be prepared to not select a tool if the evaluation process so indicates, as the typical cost of bringing a new tool into operational use is substantial.

Evaluations of candidate tools may have already been performed and be available to the organization. Such information maybe used to reduce the cost of candidate tool evaluation.
NOTE 2- Previous evaluations which have been ~ormed on a different version of the candidate tool may still yield useful information. Similarly, evaluations which addressed a different set of organizational needs may still provide useful information.

This International Standard calls for the development of several plans and reports, and implicitly,for their review by various personnel within the organization. In additio~ activitiesare required to perform the four processes outlined. The format and level of detail of the data products is left to the discretion of the organization, as is the level of effort necessary to perflormthe activities.
NOTE 3- Some organizations may need to limit the scope, detail and formality of the prcwesses to apply this International Standard within existing resource constraints.
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5

Initiation process

The evaluation and selectionprocesses require the agreement of management. In line with this agreement, a set of goals for the introduction (or enhancement) of CASE technology will be established. A set of CASE tool selection guidelines will be identified and a project plan developed. The process is shown in Figure 2.
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Figure 2- Overview of initiation process

The development of a set of realistic goals is a necessary first activity. In developing goals, both a rationale for acquisition(why acquire a CASE tool) and a general policy for acquisition (what type of tool to acquire and how to do it) should be developed.
NOTE - Goal setting activiti~ including possibly the identilcation of selection criteria, may have already been perhmed as a part of other dorts prior to formally entering the initiation process of evaluation and selection of CASE tools.

The following tasks should be performed: . Develo~iiltlou ...

Review the organization'scurrent sdware development process, determining its maturity and areas of concern. Review the current state of CASE technology and observe trends for consideration as fiture reference technology. Compare the organization's current practices to possible fhture practices if
14
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CASE tools are adopted and identifi areas of potential benefit. Identi& probable impacts of CASE tools on the organization; e.g., areas where training and education, procedure guides, and technical support are needed to effectively deploy CASE technology. ex~-ens;
.

Set overall goals (e.g., productivity improvement, quality improvement, enhanced process manageability). Define evaluation and selection constraints (e.g.,cost, schedule, resources). Quanti& and classi$ expectations (based upon goals). I po Iicv for ac~ . ..+

Identi& constraints on tool acquisition (e.g., implementation cost, schedule, other resources). Develop alternate approaches to introducing/augmenting CASE technology (e.g., buy a tool, modifi an existing tool, develop a new tool). Assess the feasibility of the various alternatives in light of organizational readiness, technical considerations, perilormancespecifications, and resources. The goals and expectations establishedhere willbe used to guide subsequent activities in the overall process and, finally, to validate the selection decision.
5.2 Establishing selection criteria

Based upon the goals and expectations developed above, selection criteria should be
established:

Decompose the high level goals into a set of selection criteria to make the (go/no go) selection decision.
NOTE 1- The selection criteria should be objective and quantitative. Each selection criterion should include some &6ned threshold specfied on which the major golno go d=ision will b ~h d~g selection.

Define the relative importance of the selection criteria.
15

IS 14653:1999 ISO/IEC 14102:1995 NOTE 2- l'he relative importance of the selection criteria will be used to determine the weights assigned to tool characteristics and subcharacteristics for evaluation.

Define the level of detail and the nature of the evaluation activities to be performed.
NOTE 3- The nature of the evaluation activities covers the methods used in collecting the data. Reference, for example, how the data are measured, collected with predefmed criteria, or based upon subjective observation.

Define the evaluation.lselection scenario to be performed (see Annex A).
5.3 Project planning and control

Based upon the goals and selection criteria which have been established for the overall evaluation and selection process, a project plan should be created and a control mechanism implemented. The plan and control mechanism should be developed in accordance with the organization's normal planning and control process, and it should contain the following:

A project team organization with assigned responsibilities,
NOTE - The skill of the evaluators will have an impact on the results of the evaluation and its applicability to the organization. The evaluation personnel should be selected with this in mind, and the skill level of evaluators should be a factor in assessing evaluation results. The evaluation team should be representative of the intended tool user group.

A set of operational goals obtained by decomposing the overall goals previously established. A set of selectionguidelines:weighted selection criteria, definition of level of detail and nature, and an evaluation and selection scenario (see Annex A). A schedule of activities and their tasks, along with an estimate of resource requirements and a cost estimate. A means of monitoring and controlling the execution of the plan. If developed, the project plan and control mechanism should be updated as the project
evolves.
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6

Structuring process

The structuring of the ewduationand selection activities can begin when a set of high level goals, selection guidelines, and a project plan are in place. The structuring process begins with a requirements definition activity which is followed by two parallel activities: the gathering of inilormation on existing CASE tools, and the preparation of a list of candidate CASE tools to be evaluated. The organization of CASE tool requirements will follow the four groups of CASE tool characteristics as outlined in clause 9. The major activities are shown in F@e
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Figure 3- Overview of structuring process 6.1 Requirements definition

During requirements definitio~ the requirements for the CASE tool are collected and organized into the CASE tool characteristics as noted in clause 9. 9.1 and 9.2 identi~ the major CASE specific characteristics; 9.3 identifies general software quality characteristics, and 9.4 identifies a set of characteristics not related to quality. A

comprehensive set of requirements is necessary to select the most appropriate CASE tool, and the structuring process provides for greater ease and repeatability in the evaluation process. Three activities are required. 6.1.1 Organizational information gathering

To be able to define a set of detailed requirements to be satisfied by the CASE tool, information about the organization should be gathered, including:
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Willingness of the organization to fblly find and implement CASE tool use. Current software engineering environment within the organization, including data describing current hardware, operating soflware, and tool use. Types of software development projects undertaken by the organization include size, domain of application. Characteristics and constraints of the target systems for which software is developed. Specific expected impacts and improvements of CASE technology on the organization.
Requirements from potential tool users and end users.

Current organizational procurement policies. This informationis necessay to ensure the tool or tools are appropriate for use within the organization, they address organizational needs, and needs perceived by their fiture users.
NOTE - This information can be gathered in a number of ways, including surveys and focus groups,

6.1.2

Requirements identification

The tool user's requirements should deal with the question of what the CASE tool should do as well as its impact on the existing environment. The following tasks should be performed in building the list of requirements:
Analyze the requirements and adjust the level of detail to which requirements are defined and measured.

Evaluate the current need for CASE tools while taking into consideration those projects where the CASE tool may initially be used. Identi@ desired methodology (e.g., process-oriented, data-oriented, objectoriented). Identifj portions of the life-cycle to be supported (e.g., plaming, analysis, design).
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Identi& required fi.mctionsof the CASE tool. Identifi required quality characteristics of the CASE tool. Check the consistency of the requirements with the previously established gods.
NOTE- Theserequirements representthetotalsetof organizational requirements.It is possiblethatno single CASE tool may satis~ all of the requirements, but that individualCASEtools may satis~ a su5cientnumlmtojustify theirusebytheorganization, whichmaycontinue to searchfortoolsto support remainingrequirements. 6.1.3 Requirements structuring

The applicability of the user needs identified in clause 9, and any others which the organization may wish to add, should be defined. The purpose of this structuring is to organize the requirements in such a way that the evaluation can proceed more effectively, The tasks include: Categorize the user requirementsin terms of the organization of clause 9, and decompose them into detailed specifications. Select characteristics and specific subcharacteristics from clause 9 which can be evaluated to determine the extent to which the CASE tool meets the detailed specifications.
NOTE 1- The extent to which a CASE tool supports or implements a specific methodology may be a critical fhctor, and shcxdd be seriously considered when selecting characteristics and subcharacteristics and weighting tlmse subcharacteristics.

Identi& weights for the characteristics and subcharacteristics.
NOTES 2- The weights are applied to the ratings determined during the evaluation as part of the selection process, and reflect the relative importance of the related selection criterion as &termined during the initiation process. 3- The assignment of weights is a subjective task which has a fundamental impact on the outcome of the entire evaluation and selection process. The assignment of weights should reflect both the organization's actual requirements and the ability of the organization to evaluate the characteristic. See hnex B fm further discussion. 4 - ISO/IEC 12119:1994 addresses quality requirements applicable to CASE tools when considered as software packages, and should be consulted as part of the requirements structuring task. It provides additional guidance on a subset of the quality requirements of ISO/IEC 9126:1991.
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6.2

CASE tool information gathering

A general search of potential CASE tools to be evaluated is undertaken based upon the requirements and selection criteria established. The activities of gathering information and identi~lng the candidate CASE tools may require several iterations to quicklyand efficientlyidentii the most promising tools for firther evaluation. For the CASE toots which appear most promising for fkther evaluation, additional and more detailed data that deal with their potential acquisition are obtained. This addhional information may help to quickly eliminate many tools, allowing attention to be focused on the remaining candidates. Information to be obtained includes: Vendor general information(e.g., business history, available support, plans & strategies). Vendor's specific product development strategy. The tool's cost (e.g., price, maintenance, modifications, training). The hardware and software required to support tool use. The hardware and software required to support final application/product use. The training required for efllcient tool use. The tool's fictional capabilities.

The tool's methodology and life-cycle support, How the tool interfaces to external systems. The number of users, existence of a user's group, the users' response to the tool.
The tool's license mechanism (e.g., floating license, multi-user licenses, cross platform licenses). 6.3 Identifying final candidate CASE tools

When the set of potential candidate tools has been identified, the final candidates for selection (those to be evaluated) maybe chosen. This is accomplished through the following tasks:
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Establish a set of high-priority or critical requirements to be met by CASE tools. Compare the user's fictional requirements with the CASE tool's fictional capabilities, supporting methodology, system environment. Compare the managerial requirements with the CASE tool's cost, available training and support. Analyze the tool vendors' user base, user response, support and business history. Identi& tools satislljhg a suficient number of high-priority or critical requirements which then become the final candidates for formal evaluation. The results of the previous tasks provide the justification for the list of candidates.
NOTE - The tasks described in this paragraph represent a "screening" of possible candidates to allow the .mganization to identi$ the candidates most likely to be acceptable, given the organization's requirements w suppliers abilities. The identification of final candidates ean be performed in parallel with CASE tool information gath~ or the two activities maybe iterated. The goal is to reduce the cost of tool evaluation by only considering a screened set of fml candidates during the evaluation process.
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7

Evaluation process

Evaluation can begin when the structured requirements have been defined and a screened set of finalcandidatesfor selectionhave been chosen. Final preparations will be made for evaluating the candidate CASE tools, including the development of an evaluation plan. The evaluation activities are then performed and documented, resulting in a profile of how each CASE tool measures up to the structured requirements. The objective is to produce the technical evaluation reports necessaxy fo~the selection process, as shown in Figure 4,
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Figure 4- Overview of evaluation process 7.1 Preparing for evaluation

To define the necessary level of detail prior to beginning evaluation activities, final preparations are necessary. Based upon the list of candidate CASE tools and the structured requirements, the following tasks should be accomplished: For each atomic subcharacteristic, define or select one or more metrics and define the details of their use.
NOTE 1- ISO/IEC JTC 1 SC7 Working Group 6 has technical repotis relating to metrics under development which may help the user of this International Standard select some of the necessaty metrics.

Set the rating levels and identi& the means by which the levels will be generated or computed.
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U 14653:1999 ISO/IEC 14102:1995 NOTE 2- A measured metric value (e.g., average lines of code per module= 274) mustthen be assigned a rating value (e.g., 1.3 on a scale of Oto 4). The means by which rating levels are obtained from measurements must be identified.

Define the assessment characteristics for evaluation establishing what is acceptable, taking into consideration the rating levels previously defined and the context of use of the product.

Identifi and schedule all actiwties which must be performed as part of the evaluation process.
NOTE 3- Activities include preparing any data sets necesuuy for the evaluation, obtaining tool documentation and an instance of the tool to be evaluated, providing evaluators any necessary training in tool use, hands-on tool use, recording of tool outputs, and analysis of results.

In some cases, a Bench Mark Test (BMT) may be a part of the evaluation process. The recommended approach for a BMT includes:
q

Identi@ the required critical tool fi.mctions. Identifi a test projector sample program to be the basis for the BMT. Develop a BMT scenario, defining inputs and expected outputs. To focus evaluation activities and provide for traceability of the evaluation

q

q

process, develop an evaluation plan which includes the information above.

7.2

Evaluating CASE tools

The software is evaluated in comparison with each of the chosen characteristics. Evaluation is a process of measurement, rating and assessment.
7.2.1 Measurement

Measurements can be made based upon information obtained by examining the CASE tool itsel~ or information about it, through the following types of tasks:

Examining the vendor-supplied documentation.
Examining the source code and other intermediate products, if available. Interviewing actual users of the sofiware.
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Viewing demonstrations and interviewing demonstrators.

Executing test cases. Applying to test projects. Examining results of previous evaluations (whether in-house, third party, or other evaluations). Petiorming a BMT on the candidate tools and analyzing the results. Measurement values may be binary, based on a continuous scale (quantifiable), or textual. There are both objective and subjective characteristics.
NOTE - Objective characteristics are those which permit independent and repeatable test or metric. Subjective characteristics are those for which no independent and repeatable test or metric exists (e.g., fitness of the user intertkce to the culture of the user).

For objective characteristics, the evaluation should be made by a repeatable procedure such that another evaluator would be able to produce the same results. During evaluation, if test cases are used, a unifortq predefine, and documented set of cases should be used. For subjective characteristics, the evaluation should be performed repeatedly by more than one person or group, who will discuss and agree upon results. The evaluation results should be recorded in a quantified manner, where possible, together with textual justification, where applicable. 7.2.2
Rating

In the rating task, each measured value is rated against the scale of values defined in the evaluation plan. Rating levels are either directly generated or computed accordhg to previously defined algorithms.
NOTE - It is possible that requirements maybe revised during the evaluation, and this may require revision of rating scales.

7.2.3

Assessment

Based upon the resulting ratings and the previously defined assessment criteri~ assess the subcharacteristics and characteristics. In accordance with the selection
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guidelines and the evaluation plan, rdngs should be aggregated up to the characteristic level.
7.3 Evaluation reporting

The end result of the evaluation activities will be an evaluation report. An evaluation report may address all tools which were evaluated; alternatively, several evaluation reports may be written, each reporting on a subset of the tools evaluated. The evaluation report should contain at least the following information:
Tool information.
q

~ ~~ H.,, !

q q q q q

q q

q

q q q

CASE tool name I CASE tool version vendor host configuration cost elements background, as appropriate part(s) of the life-cycle for which the CASE tool(s) is intended type of soilware model the CASE tool(s) is based on (e.g., waterfall model, spiral model) CASE tool soflware environment (e.g., programming language(s) supported, method supported, operating systems, possible configurations, configuration used in evaluation, minimum configuratio~ database compatibility, software of other vendors required for the environment) CASE tool ibnctions inputioutput structure target audience
Evaluation process.

The report should discuss the specific activities and tasks in the evaluation process in the detail necessary to allow the reader both to understand the scope and depth of the evaluation and to repeat the evaluation, if desired. Specific results.

Evaluation results should be provided in terms of the lowest level of subcharacteristic decomposition (normally an atomic subcharacteristic). For each subcharacteristic, the metric value measured should be given in terms of the rating level for that metric.
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Based upon the lowest level results, any aggregation should be shown so as to make clear the method of aggregation any weights used, the elements aggregated, and the level to which aggregation is perfbrmed. The result will be a profile describing the results of the evaluation in terms of scores for the characteristics of clause 9, or in terms of scores for subcharacteristics, depending
on the level of aggregation.

In cases where the report covers multiple tools, or where the results of this evaluation will be compared to those of other evaluations, care should be taken to ensure the results are provided in a uniform format which facilitates comparison (e.g., by using templates). Objective results should be provided with minimal accompanying text. Subjective results should be supported by text describing the specific reasons for the metric values assigned.
NOTE - The tiormation speoified above could be organized as follows

Evaluation process 00s1s, criteria, tools evaluated Measurement tools Tool information Test Scenario Test results and evaluation Evaluation surnmtuy
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8

Selection process

Selection can begin when the evaluation reports are complete. A selection algorithm should be defined and then applied to the results of the CASE tool evaluation efforts. A decision can then be recommended, and the recommendation validated against the original set of goals and selection guidelines. An overview ,of the process is shown in Figure 5.
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Figure 5- Overview of selection process 8.1 Preparing for selection

The selection algorithm determines how the data generated during the various evaluations are combined and compared to result in ratings for each candidate. Based upon the originalgoals and selection guidelines, a final set of selection criteria is identifiedand the basis upon which these criteria are to be assessed is defined. This definition will be based upon the aggregated evaluation assessments described in 7.2.3. The algorithm for fbrther aggregating the results, comparing the candidates, and arriving at a decisionis then defined. A discussion of selection algorithms is provided in Annex B. 8.2
Applying the selection algorithm

The evaluation results are used as inputs to the selection algorithm. Information relating the candidate tools is output. Each tool's evaluation results provide a technical summaryof each tool's characteristics, aggregated up to the level specified
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in the selection algorithm (usually the characteristic level). The selection algorithm combmes the results of the evaluations of the various tools, providing a comparison for use by decision makers. 8.3 Recommending a selection decision

When the selection algorithm has been applied, a decision maybe made to acquire a tool or set of tools. This is a management decision based upon the technical comparison provided above and additional management criteria. Such a decision would indicate that the most appropriate of the candidates has been identifiedfor selection. Alternatively, the assessment of evaluation results may show a need for additionalinformation, which may indicate that some iterat$n of previous activities is necessa~. Evaluation and selection scenarios are firther discussed in Annex A. The selection decision should be justified with a rationale which summarizes the information and logic which led to the selection. 8.4
Validating the selection decision

The final activity in the process should be the validation of the recommended selection. The original goals and selection guidelines should be reviewed and compared to the evaluation results and other data relating to the recommended selection. A check should be made to ensure that if the recommendation is accepted, the high level goals (or a sufficient number of them) will be met. It maybe found that no adequate tool exists, in which case a choice maybe made between the development of a new tool or the modification of an existing tool (within the user organization or outside), or abandoning the entire evaluation and selection process.
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9

CASE tool characteristics

The user needs which drive any evaluationand/or selectionprocess will be based upon the characteristicsand subcharactenstics described below. By defining user needs in the terms used here, assessmentsand comparisons may be made based upon a broad, common, and nearly complete set of characteristics. As discussed above, a structuring activity is required to transform the set of needs initially identified by the user into the terms provided here. The top-level evaluation categories are called characteristics. Each characteristic is subdividedinto subcharacteristics. Subcharacteristics may be firther subdivided into lower level subcharacteristics. At the lowest-level, subcharacteristics are referred to as atomic su~jharacteristics. This section defines atomic subcharacteristics in terms of their attribute%each of these will be assigneda value during the evaluation process based upon one or more metrics (see 7.1, Preparation for evaluation). It is unlikely that any user of this International Standard will need to use all of the atomic subcharacteristics given below; users should select only those subcharacteristics which have significant weight with respect to their organization's requirements. There will be cases where additional needs or characteristics, specific to a particular evaluation or selection, have to be added to those listed below; in that sense, the atomic subcharacteristicslisted below can be considered a partial list, to be augmented as necessary.
Non-atomic subcharacteristics are assigned values by aggregating the values of their component subcharacteristics, weighted as called for in the evaluation plan. This aggregation task is continued until the levels of aggregation called for in the evaluation plan have been reached. The selection algorithm is then used to combine the evaluation results of the various tools for comparison and decision. 9.1 Functionality - characteristics related to life-cycle processes.

A set of attributes that bear on the existence of a set of finctions and their specified properties to support CASE tool use as it relates to sofiware engineering life-cycle processes and activities. For those life-cycle processes referenced, the definitions in ISOAEC 12207:1995 apply.
NOTE - This section addresses CASE support for several life-cycle processes. Other life-cycle processes not addressed here are absent either because CASE took+ do not typically provide support for those processes, or because the process and/or the CASE support for it are not stable at this time.
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9.1.1

Characteristic: Management Process

A set of attributes that bear on the existence of a set of finctions and their specified properties to support the managementprocess activities. For additional attributes that bear on management, see 9.1.6 for risk management. Atomic subcharacteristics: Cost and ScheduleEstimating attributes relatingto its abMy to estimate cost, schedule and other project parameters based upon organizational inputs.
NOTE 1- For example, the Constructive Cost Model (COCOMO) and its variants.

l??. attributes relating to its ability to support user entry afid im?ysis of project planning data.
NOTE 2- This subcharacteristic is more general than the subcharacteristic above; in addition to cost and schedule data, it includes, for example, computer and other facility resources, personnel allocations, annual calendar definiton and vacation planning. AUSO included is the capability of analysis of phmning &ta, such as a critical path analysis to optimize the project plan with respect to the required constraints, and the capability of reusing/modi@ng the planning data.

ct Tr_ " attributes relating to its ability to support user entry of project activity dati-+including automated data gathering.
NOTE 3- Examples of project activity data which maybe hacked inchde completion date, funds expended, resources consumed, number of documents generated, lines of code &veloped, number of test cases completed, and number of defects detected.

J%oiect Status AnaIvsis ~ and Re_ . attributes relating to its ability to support analysis of project activities based on the data tracked and provide status reports and projections in user definable formats. Ma naeimz Process es: attributes relating to its ability to support the management of pro~esses.
NOTE 4 - Process management includes defining &tailed work items by deftig inpu~ resources, output, personnel, deadline, etc.; making work item definitions available to project members; updating work status by (manually or automatically) gathering the results of the work. Query capability is also included.

9.1.2

Characteristic: Development Process

A set of attributes that bear on the existence of a set of finctions and their specified
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properties to support the development process activities. For additional attributes that bear on the development process, see 9.1.8.
NOTE - `l'heset of dewloprned Iinctions may not be exhaustive, and additional subcharacteristics may be considered as required.

9.1.2.1

Subcharacteristic:

Modeling

A set of attributes that bear on the existence of a set of fhnctions and their specified properties to support the modeling activities which can be part of the development process.
NOTE 1- Modeling fimctions reflect the CASE tool's ability to support the identification of sotlware requirements, the expression of software design, and the transformation of requirements into design.

Atomic subcharacteristics: evel0attributes relating to its abiity to support the entry and edtiig of diagrams of types of interest to the user, and to translate between diagram types, and between diagrams and text.
NOTES 2- Diagram types are defined in ISO 5807:1985. In addition, typical diagram types inclti control flow, data flow, decompmiti~ entity-relationship, object oriented, Petri nets, state transiti~ and structure charts. 3- Rules relevant to specific diagram types should be enfbmed.

am _ " "attributes relating to its ability to support the analysis of graphical figures input to the CASE tool and extracting and storing requirements and/or design itiormation.
NOTE 4- Dia~ rma@xss are, in lYuUly eaaea, integrated with diagram dlZ+WOrS, but go beyond diagram drawers in analytical capability.

~ attributes relating to its alility to support the entry and edking of requirements specification data and checking the consistency and completeness of the requirements data against allowable specification constructs and rules.
NOTES 5- Classes of requirements data which maybe eonaidered inclu& functional, data, interfhce, quality, performance, hardware, environment, cost, and schedule requirements.

.

.

.
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~ attributes relating to its ability to support the entry and edtting of design specification data and checking the consistency and completeness of design data against allowable specification constructs and rules.
NOTES 7- Classes of &sign data which maybe considered include timctional, data, interface, quality, performance, hardware, environment, cost, and schedule information. 8 -A formal language maybe used to express requirements data.

.

.

Ion Co@ruct ModeIing attributes relating to its ability to support the ent~ and editing of i~ormation describing the types of constructs that a specification can contain, including their relationships and depiction.
NOTE 9- Types of constructs which might be modeled inclu& data structures, data flows, objects, processes and states.

~ attributes relating to its ability to simulate aspects of a system's potential operation based upon requirementsador design data available to the CASE tool.
NOTE 10- Examples of aspects to be simulated include system effectiveness (operational utility), operator intdace, and architectural performance (response time, utilization, throughput).

.

attributes relating to its ability to generate a prototype model of all or portions of a system based upon user-supplied requirements and/or design information.
~ i
NOTE 11 -P@oty@g fmtures of CASE products may sometimes be replaced by 4GL/graphical user intcrthce (GUI) tools. Such replacement requires fluent transition liom modeling to design activities and back.

an Inter&e Mo-: attributes relating to its ability to model the content aspects of human-computerinteractions and the mechanical aspects of -those interactions.
NOTE 12- Examples of content aspects of human-computer interactions are presentations (e.g., menus, screen and window layouts and report designs) and querying (e.g., text, voice, touch, icon or other inputs). Examples of mechanical as~ts include window location, size, and oolom, voice volume and pitch, and touch sensitivity).

.
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9.1.2.2

Subcharacteristic:

Construction

A set of attributes that bear on the existence of a set of tlmctions and their specified properties to support the construction activitieswhich can be part of the development process.
NOTE 1- Construction Iinctions refled the tool's ability to produce operational (e.g., executable) elements of the final system to be fielded, or to modifi an existing system. Many of the fictions in this paragraph are dependent upon a specific language or languages. Examples of such languages include programming languages, data and quay languages, graphics languages, and operating system interfaces such as job control languages. The user of this International Standard should identifj those languages relevant to the specific effort.

Atomic subcharacteristics: Me Generat iOILoattributes relating to its ability to generate code in one or more specific languages based upon design data available to the CASE tool.
NOTE 2- Typical code generation capabilities include general purpose code generation, database genemtiorq queq generati~ semen displayhnenu generation. Another form of code generation is the direct generation of executable code.

ase Schema G-at iou " attributes relating to its ability to generate database schema based upon user-supplied itiormation. ; attributes relating to its ability to generate display screens based upon user-supplied information. Ort Ge~ attributes relating to its ability to automate the development of reports to be produced by the systemunder development (as opposed to the CASE tool). ~ specific languages.
. . .

attributes relating to its ability to compile code in one or more
. .

~ attributes relating to its ability to support the entry of source code in one or more specificlanguages with syntax support provided by the editor. ~ attributes relating to its ability to support the identification and isolation of errors in a program.
NOTE 3- Typical capabilities include providing tmcebaclcs and identi$ing tbult location in terms of source code.
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9.1.3

Characteristic: Maintenance Process

A set of attributes that bear on the existence of a set of fimctions and their specified properties to support the maintenance process activities. Atomic subcharacteristics: "n " attributes relating to its ability to determine that a problem: results from a user misunderstanding, has already been resolved, is going to be resolved in the context of another maintenance actio~ or is a new problem to be resolved.
1 hbundmLL- lization; attributes relating to its ability to identi~ the portion of the software requiring modification, given the identification of a problem.

#lmDact analvk attributes relating to its ability to, for each change foreseen, identifi potential consequences of making the change. Data Reverse Erwineering attributes relating to its ability to extract information from source code which defines or describes the data elements and structures of the soflware.
NO'IE 1- Examples of typical outputs include design language code, data dictionary entries, and direet en~ of design data into the CASE tool's database.

r r n in rin ~ extract process design data from source code.

attributes relating to its ability to

NOTE 2- Examples of typical outputs include design language code, design diagrams, and direet entry of design &ta into the CASE tool's database.

. attributes relating to its ability to input existing Source Code Restructuring% source code in one or more specific languages, mod@ its format and/or structure according to defined directives (e.g., reduce size of code, reduce execution time, implementcode format standard) and output a source code file in the same language.
NOTE 3- Examples of typical capabilities are pretty printers and source-level optimizers.

" attributes relating to its ability to input existing ~OU rce Code Tra nsource code written in one or more specific languages, translate it into a different language, and output the resulting code.
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9.1.4

Characteristic: Documentation

Process

A set of attributes that bear on the existence of a set of timctions and their specified properties to support the documentation process activities. Atomic subcharacteristics: .. Text Ed 1 tmg attributes relating to its ability to edit text. lcal J?dltmg attributes relating to its ability to enter and edit data in graphical format. .. Based attributes relatingto its abiity to support user definition of forms and subsequent forms-based editing. .l?l?n attributes relating to its ability to support desktop publishing. attributes relating to its ability to support hypertext
. . .

_ formats and fimctions.

variant Handli~ : attributes relatiig to its ability to reuse the same generation of the product with limited variation.
NOTE - Examples inclu& change of objects in screen panes (e.g., logo) and adaptation to local requirements (e.g., language).

. and Doc~ 0attributes relating to its abtity to accept, store and retrieve specifications of the content, format and layout of textual and graphical data to be extracted and produced, and its ability to then extract and produce the data in compliance with a specification.
Auto matic Dat&ractlon

.

9.1.5

Characteristic:

Configuration Management Process

A set of attributes that bear on the existence of a set of functions and their specified properties to support the configuration management process activities. Atomic subcharacteristics: " attributes relating to its abdity to control access to data Access COIW2L elements.
NOTE 1- Access control inclu&s the ability to specify components to be no access, read

only,
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etc.baseduponworkgroups, orothersimilar identifier, aswellastheability tocheck-out dataelements formodification andrestrict access tothem(locking) untiltheyhavebeenupdated andchecked backin (unlocking). ~ attributes relatingto its ability to maintain a record of all modifications made to the system under development or maintenance.
NOTE 2-As design and code information is changed, includes automatically updating and keeping consistent all related information kept in the tool. . . .

~ attributes relating to its abiity to maintain records and perform management fimctions on multiple versions of a system which may share common components. ~ attributes relating to its ability to provide the user with reports defining the history, contents and status of the various configyation items being managed. ~ attributes relating to its ability to support user definition of steps required to create a version (build) of the software for release, and to automatically execute those steps. attributes relatingto its ability to automatically place data elements in secondary storage for subsequent retrieval.
NOTE 3 - Archiving typically involves long term storage of infomnation off-line for use in mmstmc@ a system which was damaged or accessing data which is seldom needed. Archiving features which may be considered include level of automation, ease of retrieval, data compression capabilities and security against both loss and unauthorized access. . .

. .

.

.

9.1.6

Characteristic: Quality Assurance Process

A set of attributes that bear on the existence of a set of fbnctions and their specified

properties to support the quality assurance process activities. Atomic Subcharacteristics: ~ attributes relating to its ability to support user en~ of qualityda~ analyzequalityda~ and generate information to support qualky management.
NOTES

1- Examplesofqualitydata include quality
36
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IS 14653:1999 ISO/IEC 14102:1995 results, fault and corrective action reports, and values of complexity metrics. 2- Includes the ability to handle quality data by variant.

~ attributes relating to its ability to support risk identificatio~ risk estimatio~ risk impact wessme@ risk monitoring and controlling.
NOTES 3 -Risks to be analpd mi@t be categorhd into,but not limited to, project risks, technical risks, and business risks. 4- Risk management suppat capabilities might include hazard analysis in critical applications such as air traffic control systems or nuclear power plant cmtrol systems.

9.1.7

Characteristic: Verification Process

A set of attributes that bear on the existence of a set of fimctions and their specified properties to support the verificationprocess activities. For additional attributes that bear on the verification process, see 9. 1.8; additional ii.mctionalcapabilities relating to verification maybe found embedded in existing development subcharacteristics. Atomic subcharacteristics: Trac~ traceability analyses. .. " "attributes relating to its ability to perform

NOTE 1- Analyses normally address information km through &sign data.

the level of requirements specifications

-fication Anal= attributes relating to its ability to petiorm analyses based upon the requirements and design data available to the tool.
NOTE 2- Specific types of analyses might include algorithm, complexity, control flow, data flow, data normalization, data use, interface, human-machine interface, range bound, and structure.

rce Code* " "attributes relating to its ability to input source code in one or more specific languages and perform analyses.
NOTE 3- Examples of such analyses include the measurements of size, calculation of complexity metrics, generation of cross-refwences, and review for conformance to standard usages.
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9.1.8

Characteristic: Validation Process

A set of attributes that bear on the existence of a set of iimctions and their specified properties to support the validation process activities. Atomic subcharacteristics: . "attributes relating to its ability to formally proof of Correctnw Techprove assertions about features or operations of the software to be validated. ure &M1vsisi attributes relating to its ability to analyze failures and trace them back to defects. attributes relating to its ability to analyze defects and trace Defect ha 1& Y them forward to failures.
. attributes relating to its ability to Resu1 t EI.WL support user entry of test cases and entry of expected test case results.

rest case and Expected

Test Case md Rmsted Result Gemratkm " attributes relating to its ability to automatically generate test cases based upon existing requirements and/or design specification data available to the tool and to automatically generate expected test case results. Test Traceabilit~: attributes relating to traceability of test activities and data.
NOTE 1- Aspects include traceability of test data to other test data (e.g., test requirements to test design to test cases) as well as traceability of test data to activities and data from other life-cycle activities (e.g., requirements specifications to test cases and test cases to source code). SOU r ce code Instrumentation: attributes relating to its ability to automatically instrument code to be tested in order that test events can be identified and recorded. JnDut inputs (e.g.,

CQ ture and Repla~ attributes
keyboard,

relating

to its ability

to capture

operator

mouse) and the extent to which such data can be edited and replayed in subsequent test cases. Yest D rivirw: attributes relating to its ability to execute and/or replay test cases. attributes relating to its ability to analyze the petiormance Run-time _ of a program as it executes.
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specified dataelements andlorcodesegments, andtimingcharacteristics.

~ sofiware reliability.

.

. .

attributes relating to its ability to analyze measures of

NOTE 3- Examples of reliability measures include measures of complexity, sotlware science attributes and the mean time between failure (MTBF):

Cover~ " - attributes relating to its ability to analyze and report on test coverage, including system coverage analysis and fimction coverage analysis.
NOllZ 4- For example, statements which were/were not executed, procedures which wereJwere not called, and variables which werelwere not accessed.

Test Procedure _menL activities and a test program.

oattributes relating to its ability to manage test

NOTE 5- For example, the ability to maintain a schedule of planned activities, capture and record the results of test activities, and generate status reports.

Ion Test~ " attributes relating to its ability to support regression testing.
NOTE 6- For example, the ability to re-run previous tests; the ability to modify previous tests to account for system andhr environmental differences (e.g., date, time).

atlc Result Checkin~; attributes relating to its ability to automatically compare expected test case results and actual test case results. . . Test StWlcal Analvsis attributes relating to its ability to statistically analyze
and report on test results.
NOTE 7- For example, percent of test cases executed and per cent of test cases passed.

.

. . _ons l?nvironmentSlmul~ attributes relating to its ability to support the simulation of a real operations environment, such as a large number of users, as well as various scenarios of use and various configurations.
NOTE 8- For example, the ability to automatically generate simulated inputs to the system beimg tested based upon received system outputs.

on Tes~

.

" attributes relating to its ability to support software
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integration activities.
NOTE 9- For example, the automatic generation of body stubs for topdown automatic generation of driver procedures for bottom-up testing. testing or the

9.2

Functionality - characteristics related to CASE tool usage.

The foUowingcharacteristicsrelate the tool to its environment and the projects it will be used to support.
9.2.1 Characteristic: Environment in which the CASE tool operates.

A set of attributes which bear on the relationship between the CASE tool and its
operational (host) environment.

Atomic subcharacteristics: . . Har&vare Charactenstlcs of TOOL " attributes relating to any hardware requirements for its use.
NOTES 1- Typical hardware items to be li include processors (includiig co-processors), main memory sti, bus type, type and sizEof peripheral storage, extension or graphics cards, input and output equipment. 2- The user of this International Standard should identi~ hardware items for which the minimal requirements may not provide adequate pertiormance, e.g., main memo~. Hardware necessary to provide acceptable performance should be identified. 3- The user of this International Standard should identi~ hardware iterns which are supported by the tool as options, e.g., input and output devices.

Soflware Environment of ToOLattributes relating to any software items required for its use.
NOTE 4- Typical software iterns to be listed include opemting systems, database management systems, languages, character sets and character codes, and communications/network packages.

. . Sdhvm R~tlon B@ . attributes relating to its ability to house and manage all relevant software engineering process information. This includes its ability to make information developed in one life cycle activity available for use during other activities, as well as its ability to provide access to this itiormation to other environment elements.
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iCd ~nvtronment "attributes relating to any geographical aspects Phys of Too~ of the development environment which will impact tool use.
NOTE 7- Comideratioms include physical and temporal separation of users and the related issues of networking considerations, on-line/off-line considerations, and reposito~ updatinghnirroring at multiple sites.

.

9.2.2

Characteristic:

CASE tool integrability.

A set of attributes which bear on the ability of the CASE tool to integrate and
interoperate with other items in its operational environment. The evaluation and selection of CASE tools is performed in the context of the software engineering environment in which the tool will be used.
NOTE - Examples of other environmental items include those given in the hardware and software environment of the tool, above.

Atomic subcharacteristics: . .. Compat1 bdltv with Environment -en ts; attributes relating to its ability to interoperate with and/or directly exchange data with hardwadsoftware environments.
NOTES 1- Examples of other software tools include word processors and other documentation tools, databases, repositories, and other CASE tools. 2- If the tool contains an interthcing capability (e.g., an application programming interface) which allows the tool to be used in&pendently of environment elements, that interface should be described. 3- The extent to which the tool conforms to standards for "openness" including data interchange formats, can be evaluated in terms of a number of existing standards, including, for example, ECMA TR 55, A Reference Model for Frameworks of Sof~are Engineering Environments, ECMA 149, Portable Common Toollhvironment (PCTE)Abstract Sped@cation, ISO/IEC 9945-1: POSLYand 1S0 9075-SQL. 4- Process rnanagemen~project management and system development fhnctions may be provided by separate, specialized tools. In such a case, the connectivity between the diffkrent tools should be considered under this subcharacteristic.
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~ attributes relating to its ability to use, process, and deliver Wormation shared by other tools or part of a repository.
NOTE 5- The

.

CASEtoolshouldbe evaluated against

Metadsw ifthe hrnework is based on a specific data model (e.g., Entity/Relationship or Object Oriented), the ability of the CASE tool to generate, manipulate or access compatible structures of the relevant data model (e.g., compatible type, constraint, attribute or relationship). I%oductdata engineering if the repositmy holds a formal data description for software ti abiliv of the CASE tool to generate, manipulate or access relevant data according to its scope. .

~hg, fictional

~ attributes relating to its alility to interact with the SEE, in particular, with other tools.
NOTES 6- Invocation of the tool maybe controlled by rules, e.g., security policies, pre- or post-invocation of other tools, allowable eoncumency and synchronization. The rules may be defined by the supported method. The level of compatibility of roles might be addressed. 7-IX&rent communication mechanisms exist in different frameworks. Communication between tools em be handled by shining data within a reposito~, by message queues between tools, by client-server approaches, m by remote pmmdure calls. For example, consider the X3.46 control integration and Portable Common Tool Environment (PCTE) approaches..

~ attributes relating to the level of the homogeneity, compatibility, and consistency of its user interface with that of the remainder of the SEE.
NOTES 8- With respect to all integration characteristics, if the CASE tool was not &veloped specifically for the framework, attention should be addressed on ways of adapting it to the framework (e.g., encapsulation). For example, an important issue is the user interface, which might not be homogeneous with those of other tools despite the fact that the tools share a emrunon tiamework. 9 -In the case that the fhunework is &tined by a specification (e.g., expressed in a language such as C or Ada), the specification of the tool interface should be comparable.

.

M&d@ metadata.

Acce.M: attributes relating to the access provided to the tool's
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9.2.3

Characteristic: Aspects of the CASE tool's application.

A set of attributes which bear on the relationship between the CASE tool and the projects to which it is applied, including the environment of its products and characteristics of those products. Atomic subcharacteristics: "attributes relating to of Tool Pr_ the set of hardware and software items on which or with which the products of the tool can be used.
NOTE 1- The level of platform support in the target environment maybe a consideration, e.g., does the CASE tool generate screens, and does it generate calls to an external Qdatfonn or environment) service to generate the screens.

ce to SWds ,- Tool Pro. attributes relating to cotiormance of the products resulting from its use to standards.
NOTE 2 - Examples include language, database, repository, communication, GUI, documentati~ development contlgumtion management, security, portability, and Mormation interchange standards.

i of A~ man . attributes relating to the application domains which the CASE tool is designed to support.
NOTE 3- Examples of application domains include transaction processing, real-time, information management, and safety critical.

.

.

attributes which would result in size Size of APD Iication ~ limitations of the application and therefore limit the tooI's applicability.
NOTE 4- Such pammetem might include lines of code, levels of nesting, size of database, number of data elements, and number of ca@uration items.

~Dorte languages.

& attributes relating to its ability to support specific

languages, data and quay NOTE 5 - Examples of such languages include pgmmmkg languages, graphics languages, tmd operating system interfaces such as job twntrol languages.

s~
databases.

. attributes relating to its ability to support specific
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~ attributes relating to the set of methods or methodologies which it can support.
NOTES 6- Examples of methods or methodologies inclu& various object oriented approaches, structured (top down) approaches, data driven approaches, and real-time extensions. 7 -A CASE tool's support for a method or methodology can be evaluated based upon the extent that the tool provi&s the specific capabilities necessa~ to implement the methodology.

."" ~ attributes relating to its ability to be used in d~fferent cultures and to generate products in terms of different countries or cultures.
NOTES 8- Examples inclu& using natural different languages, character sets, character and graphic presentation modes (left-right, top-bottom), and different date formats. 9- This subcharacteristicrnay have an influence on other hardware or software environment elements.

.

9.3

General quality characteristics

The followingcharacteristicsdescribe the quality of the tool in the terms of ISO/lEC 9126.
NOTE - Further guidance on evaluating the subcharactcristics in this section can be found in ISO/lEC 12119-1994,

9.3.1

Characteristic

Functionality

A set of attributes that bear on the existence of a set of finctions and their specified
properties. The fimctions are those which satis~ stated or implied needs.

Atomic subcharacteristics: SAM of itself. attributes relatingto its abiity to prevent unauthorized use or misuse

NOTE 1- Security may also encompass all or put of the system on which the tool is used.

~ effects.

attributes relating to the provision of right or agreed results or
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~ attributesrelating to adherence to application related legal and/or regulato~ requirements.

.

~ identified standards.

.

attributes relatingto adherence or compliance with any

NOTE 2- Examples include lmguage, database, repository, cornrnunicatiem GUI, documentatiorL development, mr@uration management, security, portability, and information intemhange standada.

9.3.2

Characteristic: Reliability

A set of attributes that bear on the capability of the software to maintain its level of performance under stated conditions for a stated period of time. Atomic subcharacteristics: ~ attributes relating to its ability to correctly store and retrieve information with a high degree of confidence. ~ attributes relating to its ability to automatically initiate a backup routine to save the current state of the process.
NOTE 1- Typically backups are scheduled at a predetermined interval by the vendor or are scheduled by the user. . .

ndling attributes relating to its ability to detect abnormal behavior, not~ the user that a problem has occurred, and properly exit or save the work to the point of interruption.
NOTE 2- `Ms may include error messages displayed to the screen and a screen directed means of either exiting or saving.

"attributes relating to its ability to maintain a specified level Fault Tolerof performance (e.g., "failsoil" or reduced capability) in cases of various faults (e.g., hardware, software, network). .. Recoverabti "attributes relating to its capability to re-establish its level of performance and recover the data directly afllectedin case of a failure, and the time and effort needed for it. 9.3.3 Characteristic: Usability

A set of attributes that bear on the effort needed for use, and on the individual
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assessment of such use, by a stated or implied set of users. Atomic subcharacteristics: . . User Fmdlmew - attributes relating to its ability to integrate into the tool usex'sactivities, taking into account the user's level of experience and expertise, and the concepts, informatio~ representations and procedures that are part of the user work domain and culture (professional and individual). attributes relating to the provisions to allow the tool user to know the status of tool operatio~ to establish the causal relationship between user actions and tool status, and to assess and direct user actions on the tool.
NOTE 1- Capabilities maybe provided in the form of on-line help fatures, and diagnostic and error messages.

. ; attributes relating to the consistency of logic within an ldom~eltv application or across applications, at the procedural level as well as for the presentation of information. Ad@&l@i attributes relatingto the abfity of its interface to adapt to various task requirements, strategies, habits, and cultural modes (e.g., languages, character sets, date formats).
NOTE 2- Adaptability has several aspects: the ability to adapt to users with differing levels of experience, the ability of the user to customize input and output methods (e.g., macros and screen displays and formats), and in the number of procedures, options and commands available to a user to achieve a given objective. ..

Clarity of Contro 1;attributes relating to the extent to which the semantics of the dialogue steps (e.g.,menu selections, window choices) used to control the tool, reflect the resulting action, and the predictability of the action. Error Han$lng "attributes relating to its abilities to help and guide the user in identifying and correcting errors, and to maintain tool integrity (avoiding incorrect data and process changes). Concisen w attributes which decrease the required number of steps to identi~ and memorize, and which increase the efficiency of the dialogue. e of Learmng attributes relatingto the amount of time and effort required for a user to understand normal CASE tool operations and to become productive.
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attributes relating to the overall quality of the TOOI Documental ion Q* documentation provided with the tool.
NOTES 5 - Documentation quality understandability, and ease of overview. factors include: completeness, correctness, consistency,

6 -To the extent that the tool implements a methodology, descriptions of the methodology should accompany the tool.

of Install~ " attributes relatingto how easy it is for the user to play the required role in initial installation and in the subsequent installation of updates.
9.3.4 Characteristic: Efficiency

A set of attributes that bear on the relationship between the level of pefiormance of
the software and the amount of resources used, under stated conditions.
NOTE - In evaluating a CASE tool against the subcharacteristics which follow, consideration should be given to jobs of both typical and maximal size.

Atomic subcharacteristics: ~ stated tasks. attributes relating to the pefiormance of the tool in performing

NOTE 1- Examples include query response time and time to analyze 100,000 lines of code. In some cases, performance benchmark data are available tlorn external sources.

table R~onse Time; attributes relating to the acceptability of the time required for the CASE tool to respond to a user input with the appropriate response in the expected operational environment. . Stor~ Req~ ` attributes relating to the amount of mass storage required (e.g., disk, tape) to accommodate the tool itself and any databases required and/or generated by the tool.
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~ attributes relating to the amount of CPU le MemoV Caddressable memory required to load and operate the tool.
NOTE 2- A determination of the amount of memo~ required for acceptable tool performance should be made, as many tools will operate in a memo~-poor environment, but will do so only marginally.

_le Process irw Sped "attributes relating to the processor (type and speed) required to operate the CASE tool at an acceptable level of performance.
9.3.5 Characteristic: Maintainability

A set of attributes that bear on the effort needed to make specified modifications. Atomic subcharacteristics: Vendor Su~Dort"attributes relating to the availability, responsiveness, and quality of services provided by the vendor to tool users.
NOTE 1- Such support scmices might include telephone support, local technical support, on-site support, publication of "known defect lists".

~bilitv of Tool to FolIow Changes in Meth* attributes relating to the ab]lity of the tool vendor to modifi the tool to maintain methodology support, as a methodology changes over time. ~ attributes relating to the vendor's track record in making regular updates which address recognized problems and/or provide additional capabilities. Fx~andab ilitvi attributes relating to its ability to be easily modified to meet expanded user needs without requiring major modification, expense, or schedule change.
NCTE 2- Related to changeability, except here, vendor intervention or addkional hardware and/or software may be required.

9.3.6

Characteristic: Portability

A set of attributes that bear on the ability of the soflware to be transferred from one environment to another. Atomic subcharacteristics:
..

lhtvto D&erent ~ware
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run on various versions of the same hardware platform or on different hardware platforms. . . With Different ODer~ attributes relating to its ability to run on various versions of the same operating system or on different operating systems, and the ease with which it can be modified to run on updates to an operating system.
Com~~djtv

. ..

To& attributes relating to the ability of one version of the tool to use data generated by a different version of the tool, and the extent of data manipulation required for reuse.
Windowing Svstem~ attributes relating to its ability to be ported between windowing systems (e.g., Open Look and Motif). 9.4 General characteristics not related to quality

to Move D@ Between Vetithe

Yortabilit?

w ith

The following characteristics are general in nature, and address both the tool itsel~ the tool developer and/or vendor:
9.4.1 Characteristic: Acquisition Process

A set of attributes that bear on the acquisition process necessary if the CASE tool is selected for adoption. Atomic subcharacteristics: Cost o f TOOI Implemutat ion ; attributes relating to the cost of implementing the tool.
NOTE 1- All aspects relevant to the speeific instance should be considered. Not only tool purchase price, but also the costs for installation, initial maintenance, hardwaredsoftware revision or upgrades, and training. Price data on all relevant eontigurations should be considered, including single copy, multiple copies, site license, corporate license, and network license.

P Pollclw attributes relating to the supplier's licensing policies.
NOTE 2- These include available license options, the right to copy (media and documentation), and any restrictions andbr f= for secomky usage. `l'hatis, the tool user sells products which include some element or aspect of a tool used to &velop the product. Also, any terms and conditions, including produet guarantee, which apply to the tool.

. .

rtR ~

. .

attributes relating to the identification of any restrictions
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on the export of the tool, or of any secondaty usage of the tool.
9.4.2 Characteristic: Implementation

A set of attributes that bear on the tool's delivery, installation and operation. Atomic subcharacteristics: cost J?~ "attributes relating to the cost of tool operation.

NOTE 1 -A costhmefit analysis maybe performed, or some consideration given to the expected level of productivity of the CASE ted.

De v elopment/De livery Constraints; attributes relating to any schedule constraints involving fitiher product development and/or delivery. In addition, the time required for the tool's users to become productive with the tool (learning curve) should also be considered. . . . Workarou@ ReWlred for User ~ " attributes relating to any workarounds which would be required to implement the CASE tool in the user's environment.An exampleof such a workaround is finding a way to use a centralized tool (single common database) in a distributed environment. ~ use.
NOTE 2- Examples include floor space, table space, fbrniture, electricity and other physical requirements generated by new tool-related hardware or tool use considerations.

attributes relating to in.tlastructure requirements for tool

9.4.3

Characteristic: Support Indicators

A set of attributes that bear on the vendor's ability to provide tool support. Atomic subcharacteristics: Ier _ overall capability.
.

attributes relating to a general indication of the supplier's

fial

NOTE 1- Examples include: the supplier's size, number of years in business, market share, a statement listing of any complementary products, identification of relevant business relationships (e.g.,othertool suppliers), local presence, and the company's planned direction for future development.

Product Prom . attributes relating to general product use information. 50

IS 146S3 :1999 ISO/IEC 14102:1995 NOTE 2- Examples include: product age, number of paid installations, number of distributors of the tool, miwence, sim and level of activity of a user's group, number of versions supported, availability of dial up help hot-line, availability of maintenance contracts, formal problem reporting ays~ product development prolead times (e.g., new !hnctions, tiouble reports, customer support), body of applications, tiee+domtiom error, and availability (commercial, govemment, public domain, in-house, or under &velopment).

~ attributes relatingto the availability of training materials and training courses, both at the vendor's facility and at the purchaser's fhcility.
NOTE 3- Conditions under which training can be supplied, including the availability of causes customized for specific user needs should be considered.

. .

.

..

9.4.4

Characteristic: Evaluation or Certification

A set of attributes that bear on the evaluation or certification of the developer or the product. Atomic subcharacteristics: . . . attributes relating to the evaluation or Deve1 OP~ or c~ certification by a professionally recognized so&are engineering evaluation organization that the developer'ssoftware engineering practices meet some minimum level, or the vendor's intention to obtain such an evaluation or certification.
NOTE - For example, a capability maturity assessment based upon the S&ware Engineering Institute's Capability Maturdy Model,orISO9001 certification.

Product CertificatioK attributes relating to the certification by an appropriate party that the tool complies with a specific standard.
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Annex A
(informative)

Considerations on the use of this International Standard
There are a number of possible scenarios when evaluating and/or selecting CASE tools. IX&rent businessgoals may be handled by the processes defined herein. The initiationprocess definedin the International Standard can be used in all scenarios to help guide the overall project plan. In planning the work the following two areas should be considered:
A.1 What is the scope of the CASE tool search?

The set of evaluation and selection processes maybe performed to satis@ one of several objectives. These include the following: The organizationwishesto decide whether or not to purchase a specific tool. There is only one candidate; the candidate should be evaluated to determine whether its benefit will justifi its purchase, and a selection decision made. The organizationwishesto provide automated support for some aspect (e.g., life-cyclephase) of its software development process. For example, it may decide it wants to provide a tool for maintainingrequirementsand top level design information, to trace requirements to desig~ and to generate related documentation. A detailed set of requirements shouldbe defined and candidates identified, a selection algorithm adopted, and the candidates evaluated. An organizationwants to improve its software developmentprocess, but is not sure where to start. The organization should review the activities and tasks in the initiation process. Before considering the applicability of a specific CASE tool, the organization should consider a more complete assessment of its current processes. This will assist the organization in its decision whether to install a CASE tool.
A.2 To what extent can existing evaluations be used?

The user of this International Standard provides consistency in the evaluation and selectionprocesses. This can be used to benefitthose instances where an organization uses one or more evaluationsthat have been done at some earlier time, either by itself or by another organization. In such a case, care must be taken to ensure that the evaluationsperformedby dfikrent organizationsare appropriate for comparison. User needs, upon which the various evaluations were based, should be compared and 52
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consideration given to the expertise, motivation, and possible biases of the organization(s) which originally performed the evaluation. In addition, the metrics used to evaluate the subcharacteristicsshould be examined for applicability. It should be noted that if new evaluation results are to be compared to results of previous evaluations, the metrics and rating scales used should be as similar as possible. In another scenario, one or more CASE tools are only evaluated without proceeding to selection. For example,the evaluation(s)maybe petiormed as a self-evaluation by a CASE tool supplier, or for entry into a data repository by a tool evaluation organization. In this case, the evaluations should be general in nature, petiormed against all relevant user needs, in as much as the needs of interest to fbture selectors cannot be determined.
A.3 Other considerations

During the evaluation and selection processes, there are other considerations to be taken into account. For example: Information may be obtained during evaluation activities which leads to a modification of the requirements and, possibly, re-evaluation of some candidates. Upon completion of evaluations, there may be no significant difference between the top candidates. This may be addressed by selecting arbitrarily, by modi~lng the selection algorithm, or by modi~lng the requirements and/or metrics and performing additional evaluation activities. Levels of evaluation may be petiormed, interspersed with selection (or elimination) activities. For example, if a large number of candidates is identified, an initial, top-level evaluation may be performed of all candidates and a costhenefits analysis performed to allow the elimination of some candidates. Further, more detailed, evaluation of the remaining candidates maybe performed, followed by the eliminationof some. This process maybe repeated several times until a final selection is made.
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Annex B
Examples (informative) of selection algorithms

An important part of the selection process is the application of some algorithm to the evaluation results. Algorithms used by organizations vary widely. Selection

processes rely upon some assignments of weights to characteristics, and then combining the resulting weighted evaluation ratings using some algorithm.

B.1 Considerations

in assigning weights

The weights assigned to subcharacteristics and characteristics must reflect the organization's actual needs. If the assignment of weights reflects the opinion of a few

irdviduals' individualpreferences rather than the organization's actual needs, there is a higher risk that the CASE tool selected will not be successfully adopted. To allow the evaluation to provide usefid information for the selection process, the weights must provide for the discrimination between candidate tools where the tools differ substantially in meeting the particular organization's needs. Experience indicatesthat characteristics which are difilcult for an organization to evaluate (e.g., lack of access to dat~ insufilcient resources) tend to result in a very narrow range of ratings for the candidate tools. For example, if an organization is not in a position to critically evaluate some characteristic, e.g., reliability, the candidate tools will most likelybe assigned ratings in a narrow range (e.g., 3.0- 3.5 out of 4). The higher the weight assigned to such a characteristic, the smaller will be the spread between the weighted scores, and thus the less the process will be able to discriminate between tools. Thus the user of this International Standard should assign the highest weights to characteristics which reflect the organization's actual needs and which can be evaluated with some degree of detail.
B.2 Types of algorithms

There are several general approaches to selection algorithms:
An organization may use an algorithm which leads to a single rating for each candidate tool and then compare the ratings.

An organization may establish upper and lower thresholds within which a
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tool's rating must fall. An organization may make a selection based upon a profile of each tool and the use of management judgement. Examples of several typical approaches are provided below. Algorithms commonly in use include cost-based algorithms, score-based algorithms, and rank-based algorithms. They are discussed below, and examples are provided. There are advantages and disadvantages to each approach -no recommendation is made. Organizations desiring an honest evaluation and selection process should choose an algorithm which it has sul%cient resources to implement, and which is appropriate to the specific case in point.
B.3Cost-based algorithms

These algorithms identii some minimalacceptable level of capability (based upon the needs) and identi~ all tools providing that capability. The acceptable tools are then ranked according to cost. The lowest cost, acceptable tool is presumably recommended. This approach is sometimesmandated by organizational procurement regulations. Proponents of a cost-based approach usually cite the ease of application, perceived fairness (more objective), and lowest resulting cost of recommended tool. Opponents of this approach ofien counter that the precise definition of the actual user requirements is very difficult, and represents a major risk. Another criticism is that this approach is not sensitive to costlbenetl tradeoffs. That is, the low cost tool will meet the minimum requirements,but a tool which is somewhat more expensive might, by exceeding the minimum requirements, provide a substantially higher level of productivity, resulting in a better overall value in the long term.
B.4 Example of the application of a cost-based algorithm

An organization decides that it wants to provide its software developers with a detailed design tool which will allow its users to enter design data and which then produces a data dictionary, certain specific charts and diagrams, and performs a number of consistency and completeness checks (all well-defined). The tool to be purchased must operate in a specific hardware/soflware environment. The organization identifies all the candidate tools which run in the desired environment and claim to provide the required capabilities. It obtains evaluation copies of the carddates, assigns evaluationpersonnelto verify that the tools do indeed provide the
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required products in a satisfactory manner. The costs of each candidate are then computed. This organization considers "cost" to include purchase price, five years of maintenanceand updates, documentatio~, and initial installation and training. The tool whose overall cost is lowest is purchased.
B.5 Score and rank-based algorithms

Score and rank-based algorithms are very similar in that in each case, a single value for each tool is calculatedby multiplying the weight given to each user need by some number and adding those products. In the case of a score-based algorithm, the
number which is weighted is a score of how well the tool meets the user need, according to some predefine scale (e.g., 4 on a scale of 1 to 5). In the case of a rank-based algorithm, the number which is weighted is the ordinal rank of how well this tool meets the need when compared to the other tools under consideration (e.g.,

second best out of five candidates). Score-based algorithms attempt to provide an absolute measure of each tool evaluated,and tools can be individually evaluated. The tool with the highest score is recommended. Rank-based algorithms attempt to provide relative measures of a number of tools; tools cannot be individually evaluated, and the result for a given tool is dependent upon the set of competing tools being evaluated. The tool with the lowest score is recommended. When comparing score and rank-based algorithms to the cost-based approach, proponents of score and rank-based algorithms contend that these algorithms are more sensitiveto the needs of tool users: evaluations are typically less fi.mctionaland more qualitative in nature, and are oflen performed by fhture users. They also contend that these approaches are more sensitive to ranges of capabilities (vs. minimum capabilities) and therefore amenable to costhenefit trade-offs and productivity enhancement analyses. Proponents of cost-based approaches oilen argue against score and rank-based algorithms,arguing that the evaluationsbecome much more expensive and subjective. They contend that evaluators allow their personnel biases to direct the assignment of scores or ranks, allegingthat evaluators decide first which tool to select (based upon subjective characteristics) and then assign scores or ranks which justi~ their earlier decision. They also contend that much of the additional detail usually provided in score and rank-based evaluations is specious; that in practice, when the weights are applied and the values from many user needs are combined, the results are usually ve~ close, and the differences in scores which are obtained are somewhat arbitrary. When comparing score-based algorithms to rank-based algorithms, proponents of score based algorithms point out that score based algorithms provide an "absolute" measure of a tool's quality, and are independent of other tools, in particular, of the
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specific set of other candidates. They identi~ as an advantage that score-based evaluations can be carried out independently of one another, using the most appropriate personnel, resources and schedule. Proponents of rank-based approaches counter that evaluatingtools in a vacuum is pointless, and that when an organization is evaluatingtools with an imminentpurchase in mind, direct comparison between the tools is required, and that independence is no advantage. They also point out that while rank-based evaluationsrequire the same personnel to evaluate all the candidate tools, it is very difficult for different evaluators to consistently apply the same evaluation characteristics, making the score for a tool a fimction of its evaluators, rather than the tool's intrinsic quality.
B.6 Profile algorithms

Consumer product testing agencies often provide results in the form of a profile of each candidate product. In the context of CASE tools, the characteristics of major importance to the user are evaluated and the results input into the selection process. No specificamalgamationof these partial results is done. The selector(s) reviews the profile, and based upon the judgement of the relative importance of the various characteristics to the organization, makes a selection.
B.7 Other applicable algorithms

There are a number of additional selection algorithms, developed in the academic arena, which might be applied when appropriate for the organization. These are particularlyusefi.dwhen an organizationis faced with evaluation data which are &zy or sparse and is having difficulties in amalgamating the viewpoints of multiple evaluators and/or selectors. Borda's algorithm (Black, 1958; Fishburn, 1973) - a sum of the ranks algorithm. Condorcet'salgorithm(IM@ 1958;Fishburn, 1973) - a pairwise comparison algorithm. Dodgson's algorithm (Blac~ measurement algorithm. 1958; Fishbu~ 1973) - a preference

Fishbum's algorithm (Fishburq 1973) - a preference ordering algorithm. Lexicographic algorithm - a criteria comparison algorithm. Analytic Hierarchy Process (Saaty, 1980) - a structured algorithm.
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The algorithmsdiscussedabove are all quantitatively based. Another approach is the quaktatively based approach referred to as Grounded Theory. Rather than starting by identi@g requirements to be met and criteria to be measured, this approach beginswithan exmhation of experience to date; e.g., discuss with CASE tool users their experience with CASE technology as a starting point. Relevant references include: Glasser & Strauss, "l'he Discovery of Grounakd l%eory, Strategies for qualitative research", Aldine, New York 1967 Bubcoko, Janis A., Jr., "Towar& a Corporate Knowledge Reposito@, SYSLAB Report No. 91-023, 1991
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